CCP porkbelly
Pator Tech School
Minmatar Republic
0
   |
Posted - 2011.11.07 10:11:00 -
[1] - Quote
Hullo.
You pose some good questions. As one of the architects of the EVE networking, erm, architecture, I think I can be of assistance:
- Connection model: There is a single client connection maintained between the client and the game server. As observed, a classic blocking model is untenable because of the sheer amount of threads that would be required for this.
Instead, we use asynchronous communications, with Stackless Python microthreads. Each microthread (or Tasklet as they are more correctly known) sees its communications as being blocking, but behind the scenes an IO scheduler thus blocks it and wakes it up again when IO is ready.
The particular implementation we use is based on Windows IO Completion ports. This does not make use of the select() system call, which would be very inefficient with thousands of connections being handled by a single process. In stead, the scheduler gets a callback on an arbitrary worker thread from the OS every time IO is ready. The scheduler then proceeds to locate the tasklet that was blocked, prepares the results for it, and unblocks it.
The end result is, from the application programmer's perspective, a blocking IO model.
- Syncronosity: The problem you describe is solved by careful application of Homer's law: Ignoring it and it goes away.
Seriously, think about the problem you describe for a bit:
"if say you hit warp and a little later you opponent in PVP hits on the scrambler."
Realize that you and your friend are sitting in different locations. You are seeing, on your monitors, different representations of the state of the game. Even if you have synchronized atomic clock displays on your wall (ignoring for a bit relativistic effects), you might be in Essex while your friend could be in Tonga. So, according to your atomic clock, you would see everything slightly before him. But even more importantly: His percection of what you are doing is also skewed. He observes your actions through server latency, and sees them long after you initiated them, as measured by your atomic clocks. And the reverse is also true! You observe your friend's actions after the fact too. So, there is a biased symmetric delay mechanic in action between the pair of you.
This makes any notion of simultaneity an feeble concept at best, and the phrase "a little later" becomes meaningless.
On a different level, your intuition about IO scheduling is right, but useless, for a different reason: There are so many levels of delays and reordering present in the physical and virtual networks, that it becomes pointless. Even assuming equal distances to both clients, the order of arrival of the network packets to the host hardware is completely arbitrary. What the network interface hardware then does to it is a black box. So is the operating system which is beyond our control. Even for a blocking model, there is absolutely no guarantee that the first packet in on the wire will cause the corresponding blocked thread to be scheduled before the other. And even if it were, there is no reason why the threads in question could not be arbitrarily rescheduled by the preemptive scheduler before they could even so much as acquire a critical section.
The above should make it clear to you that there exists no inherent order between data arriving on separate TCP connections and therefore it is pointless to pretend that there is and try to somehow maintain it.
Having said all that though, we are careful to try to maintain whatever ordering is presented to our user level program. Since we work with tasklets, explicitly scheduled microthreads, we do schedule them to be run in the order that the IO callbacks from the OS are made to our application (barring of course preemtive thread scheduling of the already executing callbacks.) And we do go to certain lengths to ensure that the tasklets are executed in a round-robin fashion to guarantee fairness to all parties.
Hope that clears things up a bit. |